QuATON: Quantization Aware Training of Optical Neurons.

Optical processors, built with "optical neurons", can efficiently perform high-dimensional linear operations at the speed of light. Thus they are a promising avenue to accelerate large-scale linear computations. With the current advances in micro-fabrication, such optical processors can now be 3D fabricated, but with a limited precision. This limitation translates to quantization of learnable parameters in optical neurons, and should be handled during the design of the optical processor in order to avoid a model mismatch. Specifically, optical neurons should be trained or designed within the physical-constraints at a predefined quantized precision level. To address this critical issues we propose a physics-informed quantization-aware training framework. Our approach accounts for physical constraints during the training process, leading to robust designs. We demonstrate that our approach can design state of the art optical processors using diffractive networks for multiple physics based tasks despite quantized learnable parameters. We thus lay the foundation upon which improved optical processors may be 3D fabricated in the future.

Swept-source Raman spectroscopy of chemical and biological materials.

Significance: Raman spectroscopy has been used as a powerful tool for chemical analysis, enabling the noninvasive acquisition of molecular fingerprints from various samples. Raman spectroscopy has proven to be valuable in numerous fields, including pharmaceutical, materials science, and biomedicine. Active research and development efforts are currently underway to bring this analytical instrument into the field, enabling in situ Raman measurements for a wider range of applications. Dispersive Raman spectroscopy using a fixed, narrowband source is a common method for acquiring Raman spectra. However, dispersive Raman spectroscopy requires a bulky spectrometer, which limits its field applicability. Therefore, there has been a tremendous need to develop a portable and sensitive Raman system. Aim: We developed a compact swept-source Raman (SS-Raman) spectroscopy system and proposed a signal processing method to mitigate hardware limitations. We demonstrated the capabilities of the SS-Raman spectroscopy by acquiring Raman spectra from both chemical and biological samples. These spectra were then compared with Raman spectra obtained using a conventional dispersive Raman spectroscopy system. Approach: The SS-Raman spectroscopy system used a wavelength-swept source laser (822 to 842 nm), a bandpass filter with a bandwidth of 1.5 nm, and a low-noise silicon photoreceiver. Raman spectra were acquired from various chemical samples, including phenylalanine, hydroxyapatite, glucose, and acetaminophen. A comparative analysis with the conventional dispersive Raman spectroscopy was conducted by calculating the correlation coefficients between the spectra from the SS-Raman spectroscopy and those from the conventional system. Furthermore, Raman mapping was obtained from cross-sections of swine tissue, demonstrating the applicability of the SS-Raman spectroscopy in biological samples. Results: We developed a compact SS-Raman system and validated its performance by acquiring Raman spectra from both chemical and biological materials. Our straightforward signal processing method enhanced the quality of the Raman spectra without incurring high costs. Raman spectra in the range of 900 to 1200 cm-1 were observed for phenylalanine, hydroxyapatite, glucose, and acetaminophen. The results were validated with correlation coefficients of 0.88, 0.84, 0.87, and 0.73, respectively, compared with those obtained from dispersive Raman spectroscopy. Furthermore, we performed scans across the cross-section of swine tissue to generate a biological tissue mapping plot, providing information about the composition of swine tissue. Conclusions: We demonstrate the capabilities of the proposed compact SS-Raman spectroscopy system by obtaining Raman spectra of chemical and biological materials, utilizing straightforward signal processing. We anticipate that the SS-Raman spectroscopy will be utilized in various fields, including biomedical and chemical applications.

Optical diffraction tomography for assessing single cell models in angular light scattering.

Angularly resolved light scattering (ALS) has become a useful tool for assessing the size and refractive index of biological scatterers at cellular and organelle length scales. Sizing organelle populations with ALS relies on Mie scattering theory models, which require significant assumptions about the object, including spherical scatterers and a homogeneous medium. These assumptions may incur greater error at the single cell level, where there are fewer scatterers to be averaged over. We investigate the validity of these assumptions using 3D refractive index (RI) tomograms measured via optical diffraction tomography (ODT). We compute the angular scattering on digitally manipulated tomograms with increasingly strong model assumptions, including RI-matched immersion media, homogeneous cytosol, and spherical organelles. We also compare the tomogram-computed angular scattering to experimental measurements of angular scattering from the same cells to ensure that the ODT-based approach accurately models angular scattering. We show that enforced RI-matching with the immersion medium and a homogeneous cytosol significantly affects the angular scattering intensity shape, suggesting that these assumptions can reduce the accuracy of size distribution estimates.

Prediction of single-cell RNA expression profiles in live cells by Raman microscopy with Raman2RNA.

Single-cell RNA sequencing and other profiling assays have helped interrogate cells at unprecedented resolution and scale, but are inherently destructive. Raman microscopy reports on the vibrational energy levels of proteins and metabolites in a label-free and nondestructive manner at subcellular spatial resolution, but it lacks genetic and molecular interpretability. Here we present Raman2RNA (R2R), a method to infer single-cell expression profiles in live cells through label-free hyperspectral Raman microscopy images and domain translation. We predict single-cell RNA sequencing profiles nondestructively from Raman images using either anchor-based integration with single molecule fluorescence in situ hybridization, or anchor-free generation with adversarial autoencoders. R2R outperformed inference from brightfield images (cosine similarities: R2R >0.85 and brightfield <0.15). In reprogramming of mouse fibroblasts into induced pluripotent stem cells, R2R inferred the expression profiles of various cell states. With live-cell tracking of mouse embryonic stem cell differentiation, R2R traced the early emergence of lineage divergence and differentiation trajectories, overcoming discontinuities in expression space. R2R lays a foundation for future exploration of live genomic dynamics.

Multiphoton imaging of the monosachharide induced formation of fluorescent advanced glycation end products in tissues.

We studied the in vitro rate of fluorescent advanced glycation end products (fAGEs) formation with multiphoton microscopy in different porcine tissues (aorta, cornea, kidney, dermis, and tendon). These tissues were treated with d-glucose, d-galactose, and d-fructose, three primary monosaccharides found in human diets. We found that the use of d-fructose resulted in the highest glycation rate, followed by d-galactose and then d-glucose. Moreover, compared to non-collagen tissue constituents such as elastic fibers and cells, the rate of tissue glycation was consistently higher in collagen, suggesting that collagen is a more sensitive target for fAGE formation. However, we also found that collagen in different tissues exhibits different rates of fAGE formation, with slower rates observed in tightly packed tissues such as cornea and tendon. Our study suggests that for fAGE to be developed into a long-term glycemic biomarker, loosely organized collagen tissues located in the proximity of vasculature may be the best targets.

Optical Diffraction Tomography and Raman Confocal Microscopy for the Investigation of Vacuoles Associated with Cancer Senescent Engulfing Cells.

Wild-type p53 cancer therapy-induced senescent cells frequently engulf and degrade neighboring ones inside a massive vacuole in their cytoplasm. After clearance of the internalized cell, the vacuole persists, seemingly empty, for several hours. Despite large vacuoles being associated with cell death, this process is known to confer a survival advantage to cancer engulfing cells, leading to therapy resistance and tumor relapse. Previous attempts to resolve the vacuolar structure and visualize their content using dyes were unsatisfying for lack of known targets and ineffective dye penetration and/or retention. Here, we overcame this problem by applying optical diffraction tomography and Raman spectroscopy to MCF7 doxorubicin-induced engulfing cells. We demonstrated a real ability of cell tomography and Raman to phenotype complex microstructures, such as cell-in-cells and vacuoles, and detect chemical species in extremely low concentrations within live cells in a completely label-free fashion. We show that vacuoles had a density indistinguishable to the medium, but were not empty, instead contained diluted cell-derived macromolecules, and we could discern vacuoles from medium and cells using their Raman fingerprint. Our approach is useful for the noninvasive investigation of senescent engulfing (and other peculiar) cells in unperturbed conditions, crucial for a better understanding of complex biological processes.

Multiline Orthogonal Scanning Temporal Focusing (mosTF) Microscopy for Scattering Reduction in High-speed in vivo Brain Imaging.

Temporal focusing two-photon microscopy enables high resolution imaging of fine structures in vivo over a large volume. A limitation of temporal focusing is that signal-to-background ratio and resolution degrade rapidly with increasing imaging depth. This degradation originates from the scattered emission photons are widely distributed resulting in a strong background. We have developed Multiline Orthogonal Scanning Temporal Focusing (mosTF) microscopy that overcomes this problem. mosTF captures a sequence of images at each scan location of the excitation line, followed by a reconstruction algorithm reassigns scattered photons back to the correct scan position. We demonstrate mosTF by acquiring mice neuronal images in vivo. Our results show remarkably improvements with mosTF for in vivo brain imaging while maintaining its speed advantage.

DEEP-squared: deep learning powered De-scattering with Excitation Patterning.

Limited throughput is a key challenge in in vivo deep tissue imaging using nonlinear optical microscopy. Point scanning multiphoton microscopy, the current gold standard, is slow especially compared to the widefield imaging modalities used for optically cleared or thin specimens. We recently introduced "De-scattering with Excitation Patterning" or "DEEP" as a widefield alternative to point-scanning geometries. Using patterned multiphoton excitation, DEEP encodes spatial information inside tissue before scattering. However, to de-scatter at typical depths, hundreds of such patterned excitations were needed. In this work, we present DEEP2, a deep learning-based model that can de-scatter images from just tens of patterned excitations instead of hundreds. Consequently, we improve DEEP's throughput by almost an order of magnitude. We demonstrate our method in multiple numerical and experimental imaging studies, including in vivo cortical vasculature imaging up to 4 scattering lengths deep in live mice.

Mapping nanoscale topographic features in thick tissues with speckle diffraction tomography.

Resolving three-dimensional morphological features in thick specimens remains a significant challenge for label-free imaging. We report a new speckle diffraction tomography (SDT) approach that can image thick biological specimens with ~500 nm lateral resolution and ~1 µm axial resolution in a reflection geometry. In SDT, multiple-scattering background is rejected through spatiotemporal gating provided by dynamic speckle-field interferometry, while depth-resolved refractive index maps are reconstructed by developing a comprehensive inverse-scattering model that also considers specimen-induced aberrations. Benefiting from the high-resolution and full-field quantitative imaging capabilities of SDT, we successfully imaged red blood cells and quantified their membrane fluctuations behind a turbid medium with a thickness of 2.8 scattering mean-free paths. Most importantly, we performed volumetric imaging of cornea inside an ex vivo rat eye and quantified its optical properties, including the mapping of nanoscale topographic features of Dua's and Descemet's membranes that had not been previously visualized.

De-scattering Deep Neural Network Enables Fast Imaging of Spines through Scattering Media by Temporal Focusing Microscopy.

Today the gold standard for in vivo imaging through scattering tissue is point-scanning two-photon microscopy (PSTPM). Especially in neuroscience, PSTPM is widely used for deep-tissue imaging in the brain. However, due to sequential scanning, PSTPM is slow. Temporal focusing microscopy (TFM), on the other hand, focuses femtosecond pulsed laser light temporally while keeping wide-field illumination, and is consequently much faster. However, due to the use of a camera detector, TFM suffers from the scattering of emission photons. As a result, TFM produces images of poor quality, obscuring fluorescent signals from small structures such as dendritic spines. In this work, we present a de-scattering deep neural network (DeScatterNet) to improve the quality of TFM images. Using a 3D convolutional neural network (CNN) we build a map from TFM to PSTPM modalities, to enable fast TFM imaging while maintaining high image quality through scattering media. We demonstrate this approach for in vivo imaging of dendritic spines on pyramidal neurons in the mouse visual cortex. We quantitatively show that our trained network rapidly outputs images that recover biologically relevant features previously buried in the scattered fluorescence in the TFM images. In vivo imaging that combines TFM and the proposed neural network is one to two orders of magnitude faster than PSTPM but retains the high quality necessary to analyze small fluorescent structures. The proposed approach could also be beneficial for improving the performance of many speed-demanding deep-tissue imaging applications, such as in vivo voltage imaging.

Stomach tissue classification using autofluorescence spectroscopy and machine learning.

BACKGROUND AND OBJECTIVES: Determination of stomach tumor location and invasion depth requires delineation of gastric histological structure, which has hitherto been widely accomplished by histochemical staining. In recent years, alternative histochemical evaluation methods have been pursued to accelerate intraoperative diagnosis, often by bypassing the time-consuming step of dyeing. Owing to strong endogenous signals from coenzymes, metabolites, and proteins, autofluorescence spectroscopy is a favorable candidate technique to achieve this aim. MATERIALS AND METHODS: We investigated stomach tissue slices and block specimens using a fast fluorescence imaging scanner. To obtain histological information from broad and structureless fluorescence spectra, we analyzed tens of thousands of spectra with multiple machine-learning algorithms and built a tissue classification model trained with dissected gastric tissues. RESULTS: A machine-learning-based spectro-histological model was built based on the autofluorescence spectra measured from stomach tissue samples with delineated and validated histological structures. The scores from a principal components analysis were employed as input features, and prediction accuracy was confirmed to be 92.0%, 90.1%, and 91.4% for mucosa, submucosa, and muscularis propria, respectively. We investigated the tissue samples in both sliced and block forms using a fast fluorescence imaging scanner. CONCLUSION: We successfully demonstrated differentiation of multiple tissue layers of well-defined specimens with the guidance of a histologist. Our spectro-histology classification model is applicable to histological prediction for both tissue blocks and slices, even though only sliced samples were trained.

Bendable long graded index lens microendoscopy.

Graded index (GRIN) lens endoscopy has broadly benefited biomedical microscopic imaging by enabling accessibility to sites not reachable by traditional benchtop microscopes. It is a long-held notion that GRIN lenses can only be used as rigid probes, which may limit their potential for certain applications. Here, we describe bendable and long-range GRIN microimaging probes for a variety of potential micro-endoscopic biomedical applications. Using a two-photon fluorescence imaging system, we have experimentally demonstrated the feasibility of three-dimensional imaging through a 500-µm-diameter and ∼11 cm long GRIN lens subject to a cantilever beam-like deflection with a minimum bend radius of ∼25 cm. Bend-induced perturbation to the field of view and resolution has also been investigated quantitatively. Our development alters the conventional notion of GRIN lenses and enables a range of innovative applications. For example, the demonstrated flexibility is highly desirable for implementation into current and emerging minimally invasive clinical procedures, including a pioneering microdevice for high-throughput cancer drug selection.

Label-free three-photon imaging of intact human cerebral organoids for tracking early events in brain development and deficits in Rett syndrome.

Human cerebral organoids are unique in their development of progenitor-rich zones akin to ventricular zones from which neuronal progenitors differentiate and migrate radially. Analyses of cerebral organoids thus far have been performed in sectioned tissue or in superficial layers due to their high scattering properties. Here, we demonstrate label-free three-photon imaging of whole, uncleared intact organoids (~2 mm depth) to assess early events of early human brain development. Optimizing a custom-made three-photon microscope to image intact cerebral organoids generated from Rett Syndrome patients, we show defects in the ventricular zone volumetric structure of mutant organoids compared to isogenic control organoids. Long-term imaging live organoids reveals that shorter migration distances and slower migration speeds of mutant radially migrating neurons are associated with more tortuous trajectories. Our label-free imaging system constitutes a particularly useful platform for tracking normal and abnormal development in individual organoids, as well as for screening therapeutic molecules via intact organoid imaging.

Label-free discrimination of tumorigenesis stages using in vitro prostate cancer bone metastasis model by Raman imaging.

Metastatic prostate cancer colonizes the bone to pave the way for bone metastasis, leading to skeletal complications associated with poor prognosis and morbidity. This study demonstrates the feasibility of Raman imaging to differentiate between cancer cells at different stages of tumorigenesis using a nanoclay-based three-dimensional (3D) bone mimetic in vitro model that mimics prostate cancer bone metastasis. A comprehensive study comparing the classification of as received prostate cancer cells in a two-dimensional (2D) model and cancer cells in a 3D bone mimetic environment was performed over various time intervals using principal component analysis (PCA). Our results showed distinctive spectral differences in Raman imaging between prostate cancer cells and the cells cultured in 3D bone mimetic scaffolds, particularly at 1002, 1261, 1444, and 1654 cm-1, which primarily contain proteins and lipids signals. Raman maps capture sub-cellular responses with the progression of tumor cells into metastasis. Raman feature extraction via cluster analysis allows for the identification of specific cellular constituents in the images. For the first time, this work demonstrates a promising potential of Raman imaging, PCA, and cluster analysis to discriminate between cancer cells at different stages of metastatic tumorigenesis.

In vivo visualization of butterfly scale cell morphogenesis in Vanessa cardui.

During metamorphosis, the wings of a butterfly sprout hundreds of thousands of scales with intricate microstructures and nano-structures that determine the wings' optical appearance, wetting characteristics, thermodynamic properties, and aerodynamic behavior. Although the functional characteristics of scales are well known and prove desirable in various applications, the dynamic processes and temporal coordination required to sculpt the scales' many structural features remain poorly understood. Current knowledge of scale growth is primarily gained from ex vivo studies of fixed scale cells at discrete time points; to fully understand scale formation, it is critical to characterize the time-dependent morphological changes throughout their development. Here, we report the continuous, in vivo, label-free imaging of growing scale cells of Vanessa cardui using speckle-correlation reflection phase microscopy. By capturing time-resolved volumetric tissue data together with nanoscale surface height information, we establish a morphological timeline of wing scale formation and gain quantitative insights into the underlying processes involved in scale cell patterning and growth. We identify early differences in the patterning of cover and ground scales on the young wing and quantify geometrical parameters of growing scale features, which suggest that surface growth is critical to structure formation. Our quantitative, time-resolved in vivo imaging of butterfly scale development provides the foundation for decoding the processes and biomechanical principles involved in the formation of functional structures in biological materials.

Structures and topological defects in pressure-driven lyotropic chromonic liquid crystals.

Lyotropic chromonic liquid crystals are water-based materials composed of self-assembled cylindrical aggregates. Their behavior under flow is poorly understood, and quantitatively resolving the optical retardance of the flowing liquid crystal has so far been limited by the imaging speed of current polarization-resolved imaging techniques. Here, we employ a single-shot quantitative polarization imaging method, termed polarized shearing interference microscopy, to quantify the spatial distribution and the dynamics of the structures emerging in nematic disodium cromoglycate solutions in a microfluidic channel. We show that pure-twist disclination loops nucleate in the bulk flow over a range of shear rates. These loops are elongated in the flow direction and exhibit a constant aspect ratio that is governed by the nonnegligible splay-bend anisotropy at the loop boundary. The size of the loops is set by the balance between nucleation forces and annihilation forces acting on the disclination. The fluctuations of the pure-twist disclination loops reflect the tumbling character of nematic disodium cromoglycate. Our study, including experiment, simulation, and scaling analysis, provides a comprehensive understanding of the structure and dynamics of pressure-driven lyotropic chromonic liquid crystals and might open new routes for using these materials to control assembly and flow of biological systems or particles in microfluidic devices.

De-scattering with Excitation Patterning enables rapid wide-field imaging through scattering media.

Nonlinear optical microscopy has enabled in vivo deep tissue imaging on the millimeter scale. A key unmet challenge is its limited throughput especially compared to rapid wide-field modalities that are used ubiquitously in thin specimens. Wide-field imaging methods in tissue specimens have found successes in optically cleared tissues and at shallower depths, but the scattering of emission photons in thick turbid samples severely degrades image quality at the camera. To address this challenge, we introduce a novel technique called De-scattering with Excitation Patterning or "DEEP," which uses patterned nonlinear excitation followed by computational imaging-assisted wide-field detection. Multiphoton temporal focusing allows high-resolution excitation patterns to be projected deep inside specimen at multiple scattering lengths due to the use of long wavelength light. Computational reconstruction allows high-resolution structural features to be reconstructed from tens to hundreds of DEEP images instead of millions of point-scanning measurements.

Tumor cell nuclei soften during transendothelial migration.

During cancer metastasis, tumor cells undergo significant deformation in order to traverse through endothelial cell junctions in the walls of blood vessels. As cells pass through narrow gaps, smaller than the nuclear diameter, the spatial configuration of chromatin must change along with the distribution of nuclear enzymes. Nuclear stiffness is an important determinant of the ability of cells to undergo transendothelial migration, yet no studies have been conducted to assess whether tumor cell cytoskeletal or nuclear stiffness changes during this critical process in order to facilitate passage. To address this question, we employed two non-contact methods, Brillouin confocal microscopy (BCM) and confocal reflectance quantitative phase microscopy (QPM), to track the changes in mechanical properties of live, transmigrating tumor cells in an in vitro collagen gel platform. Using these two imaging modalities to study transmigrating MDA-MB-231, A549, and A375 cells, we found that both the cells and their nuclei soften upon extravasation and that the nuclear membranes remain soft for at least 24 h. These new data suggest that tumor cells adjust their mechanical properties in order to facilitate extravasation.

Multiphoton imaging of the effect of monosaccharide diffusion and formation of fluorescent advanced end products in porcine aorta.

Prolonged exposure of tissues to elevated blood sugar levels lead to the formation of advanced glycation end products (AGEs), thus contributing to diabetic complications. Since the vascular system is in immediate contact with blood, diabetic effects on aorta is a major health concern. However, the relative effect of the diffusion of sugar molecular through the vascular wall and the rate of AGE formation is not known. In this study, we aim to address this issue by incubating excised porcine aorta in D-glucose, D-galactose, and D-fructose solutions for different periods. The tissue specimens were then excised for multiphoton imaging of autofluorescence intensity profiles across the aorta wall. We found that for Days 4 to 48 incubation, autofluorescence is constant along the radial direction of the aorta sections, suggesting that monosaccharide diffusion is rapid in comparison to the rate of formation of fluorescent AGEs (fAGEs). Moreover, we found that in porcine aorta, the rate of fAGE formation of D-fructose and D-glucose are factors 2.08 and 1.14 that of D-galactose. Our results suggest that for prolonged exposure of the cardiovascular system to elevated monosaccharides 4 days or longer, damage to the aorta is uniform throughout the tissues.

Single-Shot Quantitative Polarization Imaging of Complex Birefringent Structure Dynamics.

Polarization light microscopes are powerful tools for probing molecular order and orientation in birefringent materials. While a number of polarization microscopy techniques are available to access steady-state properties of birefringent samples, quantitative measurements of the molecular orientation dynamics on the millisecond time scale have remained a challenge. We propose polarized shearing interference microscopy (PSIM), a single-shot quantitative polarization imaging method, for extracting the retardance and orientation angle of the laser beam transmitting through optically anisotropic specimens with complex structures. The measurement accuracy and imaging performance of PSIM are validated by imaging a birefringent resolution target and a bovine tendon specimen. We demonstrate that PSIM can quantify the dynamics of a flowing lyotropic chromonic liquid crystal in a microfluidic channel at an imaging speed of 506 frames per second (only limited by the camera frame rate), with a field-of-view of up to 350 × 350 µm2 and a diffraction-limit spatial resolution of ~2 µm. We envision that PSIM will find a broad range of applications in quantitative material characterization under dynamical conditions.